Repository navigation
Conversation
98a2ca8 to
663de61
Compare
mtabebe
left a comment
mtabebe
left a comment
There was a problem hiding this comment.
Nice! Leaving some feedback
|
|
||
| We can only write (append) to a collection _at or beyond_ the upper. We cannot alter history before that point. | ||
|
|
||
| Individual records within a shard (as recorded at a given point in history) are _(key, value, multiplicity)_. |
There was a problem hiding this comment.
I found this slightly confusing.
When we are reading as of a timestamp, the records have no timestamp and are just (key, value, multiplicity). But records are stored with a timestamp (key, value, timestamp, diff).
So maybe we could be more precise here since you are talking about recorded
| Today, we always read and decode an entire part file. | ||
| If we stop doing that, maybe we can improve predicate push-down and even support point (key + time) lookups. | ||
|
|
||
| Claude's diagram of Parquet chunks and pages within our one row group: |
There was a problem hiding this comment.
Bit of a nit, but I personally don't like saying "Claude's diagram" here..
| Within a shard, records are organized into: | ||
| - _Batches._ Together, they cover the entire `[Since, Frontier)` time range without overlapping. Each timestamp within the range belongs to a single batch. | ||
| - _Runs._ Each batch is made up of one or more runs. Runs are an artifact of how batches are written, not a semantic separation of records. i.e. Two runs can cover exactly the same time and key ranges. | ||
| - _Parts._ Each run is made up of one or more parts. Each _part_ of a _run_ is responsible for a range of keys. |
There was a problem hiding this comment.
Parts are size capped (default 128 MB)
| - Enable `Chunk` or `Page` statistics, which gives us a page index (in the parquet footer) with stats (e.g. column min/max/nulls) for those segmentations of the part file. | ||
| - Add "Range Read" to our Blob Store interface to read from specific offsets within a file. Then, only read the row groups or pages whose stats fit the predicate. | ||
|
|
||
| ## Writers |
There was a problem hiding this comment.
I don't know where the right place to put this is but I feel like an important thing here is:
- Persist supports concurrent writers and this is important because of things like blue green, but
- within an environment the higher level control plane typically ensures that there is a single writer: e.g., environmentd owns tables and catalog state, a compute cluster owns the materialized views (outside of replicas)
So while persist linearizes concurrent appends, we actually very rarely have conflicts.
| 1. `head`, to get an outline of the latest shard state, including a pointer to the latest state "rollup" blob. | ||
| 2. `scan`, to read from the most recent state rollup through all the subsequent state changes. | ||
|
|
||
| _Shard state is recorded as a sequence of incremental updates (stored inline in the consensus database) interspersed with periodic full-state rollups (stored in blobs), |
There was a problem hiding this comment.
Unless the update is too big.
|
|
||
| In order to append to a shard, a writer must perform an atomic `compare_and_append` operation: | ||
| 1. By comparing sequence numbers, does the actual shard state match the latest shard state the writer has seen? | ||
| 2. Durably append the writer's change to the shard's state (with the next sequence number). |
There was a problem hiding this comment.
I think this is missing something like:
First we write to blob, and then write into consensus. The consensus CAS of the state is what makes it durable.
This branch has not been deployed
2 participants
I'm writing a "Persist 101" doc as a record of what I'm learning, onboarding onto Persist.
I will periodically update this PR as I cover more ground.
Maybe this belongs in Notion instead of GitHub 🤷